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Many high school seniors who plan to attend college do not enroll, and others enroll but do not return for a second 
year. Low-income students are more likely than their higher-income peers to fall off track after high school ends, 
when they often have limited connections to advisors, insufficient information about key college transition steps, 
and concerns about falling short and fitting in. Text-message-based advising is an increasingly popular strategy to 
address these challenges, including among states and districts that participate in the federal college access program 
Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). In this study about 4,800 college- 
intending seniors in high-need high schools across the country were randomly divided into two groups: one 
received their regular GEAR UP supports in the summer before and during their first year of college, and the other 
group received these regular supports with the addition of text messaging designed to support a successful 
transition to college. The study compared the experiences and outcomes of the two groups to determine the 
effectiveness of the transition messaging. 


Key Findings 


e The college transition messaging made no difference in whether students enrolled in college immediately after 
high school, nor whether they stayed enrolled in college throughout the first year. 


e Students were sent, and responded to, the messages as intended. But, students already had more information 
about college than expected and advisors potentially needed more information to support students dispersed 
across an average of 8 colleges, both of which may have been factors that limited the effectiveness of the 
messaging. 


The summer after high school and the first year of college can be challenging for students, especially for low-income 
students who may have less familiarity with college processes and expectations.! During the summer, when high 
school services have ended, students must complete multiple pre-enrollment steps such as paying deposits, 
registering for classes, and filling out housing and health insurance forms. Once college begins, access to supports 
can still be limited, with college advisor-to-student ratios often as high as 1 to 260 at four-year institutions.2 During 
this period many low-income students, including those who are the first in their family to attend college, struggle 
with new cultural norms at college and anxiety about doing well academically and socially.3 At least partly for these 
reasons, perhaps as many 
as 40 percent of low- 
income high school seniors 
who intend to go to college 
fail to enroll the following 
fall,* and as many as half of 


Exhibit 1. How Text-Message-Based Advising May Improve College 
Enrollment and Persistence 
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low-income students who 
start college leave without 
earning a credential.5 


This study’s text-message- 
based advising program 
was designed to address 
these barriers and 
strengthen students’ 
transition into 
postsecondary education 
(Exhibit 1). This kind of 
messaging had shown 
promise in a small set of 
studies as a way to raise 
the college enrollment and 


persistence rates of low-income and first-generation college students. The messages in those earlier studies 
provided reminders about important milestones, guidance on logistics, and success strategies. The earlier studies 
emphasized the importance of certain messaging features, such as connecting students to counselors for follow up 
communication, personalizing messages to students’ specific intended college, timing messaging to key decision 
points, and framing messages to help nudge students toward taking action. New messages were added in the 
current study to help students develop a positive and growth-oriented mindset, based on emerging evidence about 
the benefits of these mindsets for academic success in college. With the help of education and psychological experts 
who had developed the earlier promising strategies, this study enhanced and integrated them into a single approach 
(see Appendix A).7 


The text-message-based advising was tested in high schools participating in the federal Gaining Early Awareness 
and Readiness for Undergraduate Programs (GEAR UP) in order to fulfill anew program need. GEAR UP is a U.S. 
Department of Education (ED) program that now provides up to seven-year grants to states and local partnerships 
of districts and higher education institutions to improve the college success of students in high-need middle and 
high schools. GEAR UP grantees use a variety of approaches to provide services to eligible students,® each grantee 
choosing its specific offerings to supplement existing activities focused on increasing high school graduation rates 
and improving college preparation of low-income students.? GEAR UP activities are expected to include some 
combination of financial aid information and assistance, mentoring, tutoring, academic and career counseling, trips 
to college campuses, and encouragement to enroll in rigorous courses. In a typical year, GEAR UP grantees around 
the country serve about half a million students in more than 40 states. 1° 


In 2015, GEAR UP grantees were beginning to take advantage of a change in federal law that allowed them to extend 
services for one year beyond high school - when students might scatter to pursue different postsecondary education 
or training options.1! Texting, and the promising research supporting it, attracted the attention of GEAR UP leaders 
as a low-cost way for GEAR UP advisors, typically based at high schools, to provide students with access to 
information and support during this critical transition period.’ As interest in this approach grew, ED’s Institute of 
Education Sciences offered to provide and evaluate an enhanced messaging program to generate evidence of 
effectiveness for both the GEAR UP community and others seeking to improve college success for low-income 
students. GEAR UP directors from 16 grants across the country agreed to participate in this demonstration of text- 
message-based advising.!3 More about the study and its design is described in Box 1. 


Box 1. Overview of the Evaluation Design 
Who participated? 


e 16GEAR UP grantees across the country, and 81 of the high schools they serve. 14 
e From these schools, 4,803 high school seniors in school years 2015-16 and 2016-17 participated. Students who 
reported both intentions to enroll in college immediately after high school and interest in receiving text messages 
were included in the study.15 Based on information collected from the students: 
o 51 percent were Hispanic; 33 percent were from single-parent households; and 71 percent would be the 
first in their family to attain a college degree. 
o 54 percent reported taking at least one Advanced Placement or International Baccalaureate course in high 
school, and 80 percent expected to earn a bachelor’s degree or higher. 


How was the study conducted? 


e Within each school, students were randomly assigned by lottery either to receive regular GEAR UP services plus the 
college transition messaging (the “messaged” group) or to receive regular GEAR UP services only (the “non- 
messaged” group).16 

e The study compared outcomes for the two groups using statistical (regression) models, which took into account the 
demographic and academic characteristics of participating students and schools. The students in the messaged and 
non-messaged groups were similar on most of the characteristics measured before the lottery,!” suggesting that the 
lottery worked as planned to create two statistically similar groups. Thus, any differences in students’ outcomes can 
be interpreted as impacts of the messaging. 18 


What outcomes were measured and why? 


e College enrollment immediately after high school, 19 because the text messages began in the summer after students 
completed high school. 
e College persistence, because the text messages continued throughout students’ first year of college. The study 
examined two measures of persistence: 
o Continuous enrollment in the year after high school. 
o Continuous enrollment into the second fall after high school. 
e _ Free Application for Federal Student Aid (FAFSA) completion for the second year after high school, because securing 
financial aid is likely essential for GEAR UP students to persist in college.2° 


What data were used? 


e  Astudent survey (82 percent of students responded) to identify eligible students and describe their characteristics. 

e —_ National Student Clearinghouse and Federal Student Aid (FSA) data to measure college enrollment and persistence. 

e FSA data to measure FAFSA completion. 

e Implementation data on messages sent and delivered, collected daily from the text messaging platform and its 
provider. 

e Administrative data from GEAR UP Annual Performance Reports as well as the Common Core of Data and the 
Integrated Postsecondary Education Data System (both U.S. Department of Education data systems) to measure 
student, college, and GEAR UP project characteristics. 


STUDENTS SENT TEXT MESSAGES WERE NO MORE 
LIKELY TO ENROLL OR PERSIST IN COLLEGE THAN 
WERE OTHER STUDENTS 


The low-cost text messaging program (about $15 per student) 
addressed the informational, logistical, psychological, and behavioral 
challenges that students may face in enrolling and staying in 
college.2!,22 The messages focused on reminders about key 
milestones, positive mindset prompts, and success strategies for the 
transition period during which students are beginning and adjusting 
to college (Exhibit 2). Although the content of the text messages was 
guided by prior research,” the study team and its experts conducted 
a significant piloting effort to improve on earlier versions and tailor 
it to GEAR UP students. The piloting included focus groups and 
debrief sessions with students who were shown the messages and 
asked for feedback, and a series of rapid turnaround, small-scale 
experiments to assess whether different message presentation 
approaches worked better or worse than others. The pilots helped to 
fine-tune the messages. ?4 


@ 


Exhibit 2. College Transition Messaging 
Focused on 
Three Types of Supports 


Reminders about key 
milestones related to enrolling 
and persisting in college 


Mindset prompts encouraging 
students to believe that abilities 
can be developed through effort 
and that worries of social 
belonging will pass 


Success strategies with concrete 
tips and suggested campus 
resources where students could 
seek additional support 


The wording and delivery of the messages were important features of the program (Exhibit 3). Messages were 


nu 


action-oriented (for example, “submit the housing form, 


register for orientation”), nudging students to meet a 


milestone or seek needed support.25 Messages were also personalized, initially based on a student’s intended college 
and then on the college where the student enrolled. For example, a message might say, “Hi, Sara! Your tuition bill is 
due August 20th. Need info about tuition payment options?” and provide a link to a relevant campus resource. 
Accuracy of the personalized information was high because 86 percent of students provided the name of their 


intended college before or at the beginning of the messaging.”® 


Messages were sent automatically through a messaging platform. The automatic messages were programmed to be 
sent to students two to three times per month starting at the end of high school and continuing through the first 


Exhibit 3. Sample College Transition Messages 


Reminder about key milestone 


(1/2) Hey Monica! Have you signed 
up for the UC Santa Barbara 


onentstion? Are midterms coming up in 


Strategy for success 


the next couple weeks? Writing 
out a study schedule can help 
you stay on track to prepare 
for your exams/papers. 


(2/2) Need to register? Visit 
http://orientation.sa.ucsb.edu/fresh 
man-orientation for more 
information. Text back if I can help. 


Encouragement for a positive mindset 


Hi Robert! How are you feeling about transitioning to college at the end of the 
summer? Reply 1-10, 1=anxious, 10=excited. 


If 7 OR ABOVE: Great to hear! Some students go back and forth between 
being excited and nervous. That’s totally normal when you’re going into a 
new situation. 

If 4-6: That’s OK, it’s normal to be a bit nervous. Most students find over time 
that their nervousness goes away. Text me if you want to talk. 

If 3 OR BELOW: It’s normal to feel nervous. Most students find that they feel 
less nervous over time. Let me know if you want to talk. 


year of college (June to 
May).27 In total, 

37 messages were 
programmed to 
correspond with key 
college milestones; for 
example, before 
registration deadlines, 
tuition payment due dates, 
and the opening of the 
Free Application for 
Federal Student Aid 
(FAFSA) renewal period. 


In addition to the 
programmed messages, 
advisors could write and 
send their own text 
messages on an ad hoc 
basis to individual 
students or groups of 
students. The messaging 
platform allowed two-way 
communication, so 
students could reply to 
receive additional support 


from a GEAR UP advisor. Advisors received training to prepare them for this role. A national college advising 
organization with experience delivering text-message-based advising provided four 1-hour webinars to GEAR UP 
advisors on the content of the text messages, key college advising topics, and best practices and strategies for 
advising via text messages. The webinars were held at strategic times throughout the messaging period so that 
information provided could focus on upcoming issues students might face, such as signing up for orientation and 
paying for college in the summer webinars, adjusting to college courses during the fall webinar, and activities 
students might need to complete soon in the winter webinar. During the first webinar, the text messaging platform 
provider instructed advisors on how to use the platform. Additional ongoing technical support was available to 
advisors throughout the text messaging program.?8 


The college transition messaging did not increase college enrollment in the fall after high school. Because 
being accepted to college is just one step toward successful enrollment, the messages reminded students about 
activities they needed to complete during the summer before the start of classes. Eleven (11) college transition 
messages were sent to students over the summer. 


However, the potential of this text-message-based advising approach to get low-income college-intending 
students through the door on the first day of college was not realized.29 About 66 percent of students in both the 
messaged group and the non-messaged group enrolled in college immediately after high school as they had 
intended (Exhibit 4, panel 1).3° This is nearly the same rate of immediate college enrollment as for low-income 
students nationally: among the graduating class of 2016, 65 percent of low-income students enrolled in college 
immediately after high school.! The lack of an effect on immediate college enrollment was consistent for most 
groups of GEAR UP students and high schools (“subgroups”) that the study examined. #2 


The college transition messaging did not affect whether students were enrolled throughout the first year of 
college. Students continue to face challenges after they enroll in college and may drop out or fail to return for a 
second year because of doubts about whether they belong in college or can meet the demands of college-level 
work. Because the GEAR UP program had been extended to help students during this period, messages 
continued throughout the first year of college rather than ending after the summer. Messages during the first 
year of college focused on keeping students in college by directing them to on-campus resources, reminding 
them of deadlines throughout the year, and assuring them that struggles with social belonging and feelings of 
fitting in are normal. 


Even though students and advisors continued to engage with each other throughout the school year, students in 
the messaged group were no more likely than those in the non-messaged group to be continuously enrolled in 
college throughout the first year after high school (Exhibit 4, panel 2).23.34 Similarly, there were no differences 
in continuous first year enrollment for most subgroups that the study examined.*> 


Exhibit 4. Impact of College Transition Messaging on Enrollment 


Panel 1: Immediate College Enrollment Panel 2: Continuous Enrollment During Year After High 
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Notes: Data include 2,819 students in the messaged group and 1,984 in the non-messaged group. Percentage of students represents those 
who (panel 1) enrolled in college on October 1 of the first year after high school and (panel 2) were enrolled continuously during the first 
year after high school (that is, between July 1 and June 30 without a break of five consecutive months or more). The effect of messaging is 
estimated using the study’s statistical model. 


Source: Federal Student Aid 2017, 2018, 2019; National Student Clearinghouse 2018. 


The college transition messaging did not prompt more students to complete the FAFSA for the subsequent 
year or continue enrollment into a second year of college. One reason students may not continue into a 
second year of college is difficulty funding their education. Renewing the FAFSA provides access to federal 
financial aid and often state and local aid, as well. Messages starting in late fall reminded students to take steps 
to finance their next year of college by completing the FAFSA. Somewhat more than half of GEAR UP students in 
both the messaged group and the non-messaged group completed the FAFSA for the second year after high 
school (Exhibit 5, panel 1).3° There were generally no differences in FAFSA completion for most subgroups 
examined.?7” 


In addition to reminders about completing the FAFSA for the next school year, the text messages sent to 
students in the spring of their first year in college encouraged them to believe in their abilities and reminded 
them to register for classes for the next school year. Yet, consistent with the results for FAFSA completion, 
students in the messaged group were no more likely to be continuously enrolled into the second year of college 
than those in the non-messaged group (Exhibit 5, panel 2).38 Similarly, there were generally no differences in 
continuous enrollment into the second fall for most subgroups examined.*? 


Exhibit 5. Impact of College Transition Messaging on Persistence into the Second Year after High School 


Panel 1: FAFSA Completion for the Second Year Panel 2: Continuous Enrollment into Second Fall After 
High School 
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Notes: Data include 2,819 students in the messaged group and 1,984 in the non-messaged group. Percentage of students represents those 
who (panel 1) completed the FAFSA by October 1 of the second year after high school or (panel 2) were enrolled continuously during the first 
year after high school (that is, between July 1 and June 30 without a break of five consecutive months or more) and were also enrolled on 
October 1 of the second year after high school. The effect of messaging is estimated using the study’s statistical model. 


Source: Federal Student Aid 2017, 2018, 2019; National Student Clearinghouse 2018. 


MESSAGING WENT AS PLANNED, BUT THE INFORMATION STUDENTS AND ADVISORS HAD 
MAY HAVE BEEN AMONG THE FACTORS THAT LIMITED ITS EFFECTIVENESS 


The study’s college transition messaging was modeled on similar programs that demonstrated promise elsewhere, 
though only for certain groups of students. The study messaging expanded on those programs by adding positive 
mindset messages derived from other successful strategies and by extending the length of the advising into and 
through the first year of college. Examining how it was carried out may provide clues about why the outcomes of this 
study varied from earlier ones, and whether programs like it could be more helpful in the future. 


e The text messages were sent as planned. One potential reason why college transition messaging might not be 
effective at increasing college going could be problems with the messaging system. If students in the messaged 
group were not sent the messages, then they would not have access to the intended advising. 


However, in this study most messages were sent to most students. Each of the messages was sent to more than 
75 percent of students (Exhibit 6).4° The minor fluctuations in messages sent were due to students’ opting out, 
phone numbers becoming invalid, and advisors tailoring which messages were sent to students based on their 
prior advising activities.*! For example, when advisors knew that a student had already completed a milestone, 
such as filing the FAFSA, the advisor could turn off the reminder message.*? This level of customization allowed 
the messages to be appropriate for each student’s circumstances. Even when students’ college plans changed, 
they were still sent some of the messages. For example, a reduced set of messages was sent to the 6 percent of 
students who changed their plans and indicated they were not going to start college during the period of the 
texting study or intended to start in the spring (rather than the fall).*3 


Messages were not sent to a small set of students who opted out of the messaging (4 percent) and to those 
whose phone number was or became invalid (3 percent). The number of students who opted out or had an 


invalid phone number is similar to other texting efforts.44 Overall, 56 percent of the 2,819 students in the 
messaged group received the full set of 37 messages. 


Exhibit 6. Each Message Was Sent to More Than Three-Quarters of Students in the Messaged Group 
100% 


80% 


60% 


40% 


Percentage of students 


20% 


0% 
First Last 
Message number 


Notes: Data include 2,819 students in the messaged group. Percentage of students represents those who were sent each text message. 


Source: Program monitoring data. 


Students were at least as responsive to the text messages as in other similar efforts, and back and forth 
communications occurred between students and advisors. It is estimated that teenagers send and receive 30 
text messages per day,*° and text messages are becoming a more common channel for colleges to reach their 
students. Thus, it could be that students sent the study’s college transition text messages did not pay attention 
to the messages, were not interested in engaging with them, or were receiving similar messages from their 
colleges. 


However, few of the colleges where the majority of students in the study enrolled provided systematic 
information to students through text messaging during the study period.* Therefore, colleges themselves 
appear to be an unlikely source of text overload for students in this study. 


The study’s findings also suggest that students were responsive to the text messages and advisors engaged with 
them through this communication channel. Eighty-four (84) percent of students responded to a message at least 
once.‘ This rate of engagement is higher than that in most of the rigorous studies of text-message-based college 
advising conducted either before or subsequent to this study. To date, 14 rigorous studies have examined the 
impact of text-message-based college advising on students’ college enrollment or persistence. Of these studies, 
12 reported detailed information on student engagement and, in all but one, the percentage of students who 
responded at least once was lower than found in this study.*® On average, the GEAR UP students responded five 
times throughout the summer and school year.*? It appears that students generally appreciated the information. 
Across messages, the most common words in students’ responses included “thank [you]” and “okay.” For 
example, common words in student responses to one of the programmed text messages about financial aid 
options are displayed in Exhibit 7.5° 


Exhibit 7. Advisor-Student Text Exchange From 
Message on Financial Aid Options 


Hi, Camila! It’s your advisor, 
Luis. Does your college plan 
feel affordable? Loan 
options and financial aid 
can be confusing. Text back 
with any questions 
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Notes: Data include 206 students in the messaged group who 
responded to this text message. 


Source: Program monitoring data. 
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Further, advisors engaged with their students through 
text messages.°! Advisors responded to messages that 
students sent,>2 and they proactively sent students text 
messages that were not direct replies to student 
messages. Eighty-seven (87) percent of students 
received at least one non-programmed message from 
their advisor.5? This level of interaction between GEAR 
UP advisors and students in the messaged group 
during the summer and first year after high school 
likely contrasts with the level of interaction between 
GEAR UP advisors and students in the non-messaged 
group. About 28 percent of students in the non- 
messaged group had access to GEAR UP advisors after 
high school only when the students initiated contact. 
These students’ advisors were not otherwise providing 
ongoing, systematic outreach to students. For the other 
72 percent of students in the non-messaged group, 
advising in the year after high school took the form of 
periodic email or phone outreach from advisors as well 
as some on-campus events organized at some colleges 


enrolling larger shares GEAR UP students.54 


The text messages were provided by advisors from GEAR UP high schools who were likely familiar with the 
students. If students receiving messages are disconnected from the advisors sending them, students may not 
pay attention. Other researchers have suggested that familiarity with the advisor on the other end of the text 
message is important to students and is most effective.®> 


However, most of the advisors engaged in the text messaging program likely interacted with the students at 
some point while the students were in high school. As many as two-thirds of the advisors in this study were 
based at GEAR UP high schools, and most had worked at the students’ high schools before the study.*°® 
Exploration of the effectiveness of the text-message-based advising did not show any differences by whether the 
advisors were from the students’ high schools or located elsewhere; for example, a district or state office or a 
college.5’ The college transition messaging in this study did not appear to be any more effective when advisors 
were familiar staff. 


As students dispersed after high school, knowledge about multiple colleges was required to support them. 
Advisors may not have the knowledge needed to adequately support students as they face challenges after they 
leave high school and once they are at college. When advisors support students starting at different colleges, the 
advisors need to know college-specific information such as registration deadlines, tuition deadlines, orientation 
processes, and academic support resources. 


Indeed, the students in the study that advisors were supporting dispersed to an average of 8 different colleges, 
with advisors supporting an average of 40 students. Though the pre-programmed messages contained college- 
specific information located and entered by the study team, the messages did not contain all the college-specific 
information that students may have needed. For example, messages contained information about the timing of 
orientation, but not information to guide course selections. To help fill the gap, a handbook provided guidance 
on where advisors and students might find college-specific information and the study offered webinar trainings 
for the advisors, as previously noted. (Appendix Section A.2 provides more information on the advisor training.) 


Advisors from 81 percent of the participating GEAR UP grantees attended all four webinars, but this training 
may not have been sufficient to fill gaps in the advisors’ knowledge. In other studies of text-message-based 
advising, advisors were often college staff and students or trained, professional college advisors.*8 In fact, West 
Virginia’s Txt 4 Success program initially had high school counselors provide text-message-based advising but 
later switched to college counselors because the latter had more relevant knowledge regarding the college 
transition.5? Only two grantees in this study had college-based advisors. There is no evidence that the text- 
message-based advising was more effective for the small share of students with college-based advisors (6 


percent of students in the study). But it is possible that the case load of colleges or students each advisor 
managed was too large to handle well or that advisors had too many competing job responsibilities. 


GEAR UP students in both the messaged and non-messaged groups already had some awareness of key 
college topics as they left high school. Students with knowledge about Key college enrollment milestones and 
expectations once on campus may not benefit from the light-touch advising offered through text messages. 
Some prior research suggests that the students who benefit most from texts on college going milestones are 
those with limited access to college application and enrollment information. 


Students in this study indicated that they had already laid some ground work for further education before any 
text messages were sent after high school. Across both students who received messages and those who did not, 
many reported they had already taken some key steps toward college enrollment: 94 percent of students had 
applied to college and 28 percent had paid a college deposit. Also, most students reported an awareness of the 
FAFSA, with 95 percent reporting that they planned to complete, or had already completed, the FAFSA as of May 
of their senior year. 


Indeed, most GEAR UP students in both the messaged and non-messaged groups reported meeting with a 
college counselor or advisor to discuss a variety of topics related to the college application process, some of 
which were also covered in the messages, such as financial aid.®! Nationally, in the 2012-13 school year, 66 
percent of low-income students who did not participate in a federal college access program met with a high 
school advisor to discuss college admission, compared to 88 percent of the GEAR UP students in the non- 
messaged group in this study (Exhibit 8).® Similarly, only 56 percent of low-income students nationally who 
did not participate in a federal college access program discussed applying for financial aid with their high school 
advisor, whereas 82 percent of the students in the non-messaged group talked with high school staff about 
financial aid. The timing and question wording differed between this study and the national surveys. However, 
the results suggest that GEAR UP students might have had greater access to general information about college 
and a higher likelihood of meeting with a college advisor while in high school than expected by the study and 
more than students in other texting studies or their low-income peers nationally. This may have dampened the 
potential impact the message-based advising could have. 


Exhibit 8. Most Students Discussed Important College Topics With 
Counselors at Their High Schools 
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Notes: Percentages represent the share of non-messaged group students who reported 
discussing the topic at least once with a college advisor while in high school. 


By topic, the data include: 
Admissions requirements: 1,923 non-messaged group students. 
Timelines for applying to college: 1,915 non-messaged group students. 
How to complete the FAFSA: 1,915 non-messaged group students. 


Source: Student survey 2016 and 2017. 


How this Study Contributes to the Accumulating Evidence about 
Text-Message-Based College Advising 


Early studies of low-cost text-message-based advising generated enthusiasm because of their potential to help improve 
college access and affordability. This prompted the current study, as well as large-scale applications of text-message-based 
advising, such as efforts tied to the Common Application and from the College Board,®3 and other studies to better understand 
whether and in what contexts text-message-based advising could be effective. The activities mirrored broader interest in 
learning how to apply behavioral and psychological insights about what prompts people to make decisions and take actions to 
improve a wide range of local and federal programs. &4 


More recent efforts suggest that text messaging is not a simple or consistent solution to help more college-intending students 
enroll and stay in college.6> The accumulating evidence indicates that text-message-based advising is only effective in a small 
set of the situations in which it may be used.® This increasingly rich body of research, to which the current study contributes, 
raises some potential considerations for organizations interested in this approach: 


Targeting of students. Among 14 rigorous studies so far, only one has found positive effects on college enrollment or 
persistence for students overall (Exhibit 9).°7 In five of these studies there were positive effects for some groups of 
students, most often students with the least knowledge of college going. The GEAR UP students in this study seemed to 
be more knowledgeable about critical steps for college enrollment and success than assumed from the start. Future 
efforts that use text-message-based advising may be more likely to be effective if they focus on students with little access 
to college counseling and information on how to enroll and succeed in college. 


Exhibit 9. Summary of Accumulating Evidence on the Effectiveness of Text-Message-Based Advising 


1 of 14 Studies Showed Overall Impacts on 
@ Students’ College Enrollment and Persistence 


Significant Overall 
Improvement 


No Significant Overall = — Se = 
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Significant impact for 
certain groups of students 


Notes: Each icon represents a single study. The list of studies and more information on the impacts found in each are provided in 
Appendix Section C.3. 


Targeting of messages to key barriers. Text messages may not be sufficient to help students overcome the biggest 
challenges to getting into and staying in college. For example, college costs are often cited by students as a very 
important factor in their decision making.®® Simply connecting students to the FAFSA may not be sufficient to improve 
enrollment and persistence, especially if students’ financial aid needs are complex. Although this study’s college 
transition messages included eight messages focused on affordability (of 37 messages in total), understanding loans and 
borrowing was not addressed. If affordability is the biggest barrier to college success then a text messaging program 
more narrowly focused on college costs and affordability might be most effective. 


Providing in-depth college-specific information to advisors. Every college has its own processes and organization, 
which could make it difficult for advisors not familiar with the college to provide useful guidance to students beyond 
information about deadlines and more generic college processes. Having advisors support students at only one or two 
colleges and providing the advisors with extensive training on each college could be beneficial. 


This study underscores the importance of repeated replications to build an evidence base for an approach that initially shows 
promise. Researchers continue to adapt and adjust technology-based advising as new information emerges, because it remains 
a low-cost approach that feels like it holds promise for helping students. 
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ENDNOTES 


1 Arnold et al. 2009; Avery, Howell, and Page 2014; Castleman and Page 2014; Castleman, Page, and Schooley 2014. 


2 Carlstrom and Miller 2013. The median advisor-to-student ratio at community colleges is even higher, at 1 advisor 
to 441 students. 


3 Arnold et al. 2009; Bettinger, Long, Oreopoulos, and Sanbonmatsu 2012; Castleman, Arnold, and Wartman 2012; 
Dynarski and Scott-Clayton 2006; Page and Scott-Clayton 2016; Roderick et al. 2008; Walton and Cohen 2011. 


4 Castleman, Page, and Schooley 2014. 

5 Nichols 2015. 

6 Castleman and Page 2015, 2016; Castleman et al. 2017; Page, Castleman, and Meyer 2020. 
7 Walton and Cohen 2011; Yeager, Walton, and Cohen 2013. 


8 Serving low-income students is the focus of GEAR UP. The program funds projects across the country to prepare 
low-income students in middle and high school for college (https://www2.ed.gov/programs/gearup/index.html). 


°GEAR UP requires grantees to provide financial aid information; encouragement for enrollment in rigorous and 
challenging coursework; activities to increase high school graduation, college applications and college enrollment; 
and for state grantees only, scholarships to students enrolling in college (HEA 20 U.S.C. 1070a-24(a)). GEAR UP 
permits grantees to provide a wide array of services and activities such as mentoring, tutoring, academic and career 
counseling, rigorous courses such as Advanced Placement or International Baccalaureate, dual enrollment 
programs, college application assistance, financial literacy and counseling to students and parents, and staff 
development (HEA 20 U.S.C. 1070a-24(b)). 


10 Author calculations based on GEAR UP 2017 Annual Performance Reports data. 


11 The Higher Education Act, as amended and reauthorized in 2008, allows GEAR UP grantees to provide services for 
seven years, “through the student’s first year of attendance at an institution of higher education” (20 U.S.C 1070a- 
21-1070a-28). 


12 West Virginia’s GEAR UP program was the first to attract attention and excitement among the GEAR UP 
community about the potential for text-message-based advising. The West Virginia GEAR UP program sponsored a 
text messaging program to support high school students transitioning into college (Castleman and Meyer 2020). 


13 Grantees from the 2011 fiscal year and onward were eligible to apply for funding to extend services for a seventh 
year, which would allow them to support students through the year following high school. The study recruited 
grantees from the first cohort of grantees eligible to provide these seventh-year services. 


14 All 80 GEAR UP grantees funded with fiscal year 2011 funds and serving 2015-16 or 2016-17 high school seniors 
were eligible to participate in the study. Of them, 16 volunteered. These 16 grantees served 282 high schools, of 
which 81 volunteered. 


15 The study’s survey was administered during the spring of students’ senior year before messages were sent to any 
students. Students were asked whether they intended to enroll in college in the fall after high school. To be eligible 
for the study, students had to answer yes and to provide a cell phone number to enable the study to send at least one 
message to all students in the messaged group. Appendix Section B.1.1 provides more details on the study sample. 


16 The lottery resulted in 2,819 students in the messaged group and 1,984 students in the non-messaged group. See 
Appendix Section B.1.2 for more details on the lottery process. 


17 As expected, students in the messaged and non-messaged groups had similar characteristics before the lottery, 
except that students in the messaged group were about 2 percentage points more likely to be Hispanic and about 3 
percentage points less likely to have taken one or more AP or IB courses (see Exhibit B.5 in Appendix B for details). 
This small set of differences is about what would be expected to occur by chance, given the number of 
characteristics examined. Even so, these differences were taken into account in the statistical models that estimate 
the impact of the college transition messaging. Appendix Section B.3.2 describes the study’s statistical model. 


18 To determine whether the difference between groups was statistically meaningful, the study used a probability 
threshold (called the p-value) of .05, a level used by most researchers. Meeting this threshold means there is a 95 
percent likelihood that the difference found was not due to chance. See Appendix B.3 for a more detailed description 
of the analyses that were conducted. 


19 Immediate college enrollment was measured as enrollment on October 1 in the fall after students’ expected high 
school graduation. Appendix Section B.3.1 contains more details on how the outcomes were measured. 


20 The study did not examine FAFSA completion for the first year after high school because many students 
completed the FAFSA for their first year while still enrolled in high school, prior to the beginning of the study. 


21 The projected cost to grantees, should they choose to send college transition messages in the future, is $15.00 per 
student. The per student cost was calculated as the total cost to access the message platform for 12 months divided 
by the number of students sent messages. Access to the message platform allowed for programming, sending, 
responding, and monitoring messages. Advisors’ time to respond to students’ messages or customize messages 
based on the colleges students attend is not considered a cost for the college transition messages because the GEAR 
UP advisors could shift how they spend their time to incorporate the addition of the messaging into their 
responsibilities, rather than hiring additional advisors. 


22 Academic preparation and costs also can be hurdles to low-income and first-generation college students’ success; 
however, the college transition messaging did not target either of these barriers. The timing of the college transition 
messaging was too late to substantially improve students’ academic preparation because it began after high school 
ended. The college transition messaging included information about applying for and receiving financial aid but 
there was not a separate financial support component. 


23 Bettinger and Baker 2014; Castleman and Page 2015, 2016; Page, Castleman, and Meyer 2020; Oreopoulos and 
Petronijevic 2018; Walton and Cohen 2011; Yeager, Walton, and Cohen 2013. 


24 See Appendix A for more details about how the college transition messages were adapted and expanded as well as 
for details about pilot testing. 


25 The full content of the text messages can be found in Appendix Section A.1. 


26 See Appendix Section A.1.2 for additional details about the alignment of students’ college intentions and actual 
enrollment. 


27 The timing and content of messages varied based on where students intended to and then enrolled in college. For 
example, the 3 percent of students who indicated that they were starting college in the spring (rather than the fall) 
received a pared-down set of messages starting later in the fall. 


28 Appendix Section A.2 provides more details on training for advisors. 


29 Additional analyses, known as sensitivity analyses, showed that these results were consistent even using 
alternative ways of measuring effectiveness. See Appendix Exhibit C.2 for results from the sensitivity analyses. 


30 Immediate college enrollment was measured as enrollment on October 1 in the fall after students’ expected high 
school graduation. 


31 National Center for Education Statistics 2017. 


32 The exception was among students from rural high schools, for whom there was a statistically significant (p<.05) 
difference: students in the messaged group enrolled at higher rates (a positive “impact”) than did students in the 
non-messaged group. Given the number of subgroups examined, however, these results may not reflect a real 
difference because conducting a large number of comparisons increases the likelihood of a result being due to 
chance. Subgroups explored include (1) first generation, (2) gender, (3) college intentions (four-year/two-year), (4) 
high school locale, and (5) high school FAFSA completion rate. See Appendix Section B.3.1 for more details on the 
subgroups explored and Appendix Exhibit C.1 for more details on the results of the subgroup analyses for immediate 
college enrollment. 


33 Continuous college enrollment takes into account both whether students enter college and whether they persist. It 
is defined as a record of whether or not a student was enrolled in any postsecondary institution following high 
school between July 1, 2016 and June 30, 2017 (for cohort 1) or July 1, 2017 and June 30, 2018 (for cohort 2) with a 
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break of less than five months. This definition aligns with measures of continuous enrollment used in other research. 
For example, Warburton, Bugarin, and Nufiez (2001) investigate how high school preparation could help first- 
generation college goers be prepared for and persist in college, defining continuous enrollment as having no breaks 
in enrollment of more than four months. Horn (2009) investigates whether community college students are on track 
to complete degrees, defining continuous enrollment as having no break in enrollment of five or more months. 
Continuous college enrollment into the second fall after high school is defined as a record of whether or not a 
student was enrolled in any postsecondary institution during the year after high school and enrollment is 
documented in a college on October 1, 2017 (cohort 1) or October 1, 2018 (cohort 2). 


34 Additional analyses, known as sensitivity analyses, showed that these results were consistent even using 
alternative methods of measuring the impact of the college transition messaging. See Appendix Exhibit C.6 for 
results from the sensitivity analyses. 


35 The one exception was a statistically significant (p<.05) negative impact for students from high schools with lower 
than average FAFSA completion rates; at these high schools, students in the messaged group were less likely to be 
continuously enrolled than were students in the non-messaged group. Given the number of subgroups examined, 
however, these impacts may not be real, because conducting a large number of comparisons increases the likelihood 
of a result being due to chance. See Appendix Exhibit C.5 for more details. 


36 Additional analyses, known as sensitivity analyses, showed that these results were consistent even using 
alternative ways of measuring effectiveness. See Appendix Exhibit C.8 for results from the sensitivity analyses. 


37 There were statistically significant (p<.05) negative impacts on FAFSA renewal for female students and students 
from non-rural high schools. Given the number of subgroups examined, however, these impacts may not be real, 
because conducting a large number of comparisons increases the likelihood of a result being due to chance. See 
Appendix Exhibit C.7 for more details. 


38 Additional analyses, known as sensitivity analyses, showed that these results were consistent even using 
alternative ways of measuring effectiveness. See Appendix Exhibit C.10 for results from the sensitivity analyses. 


39 There were statistically significant (p<.05) positive impacts on continuous enrollment into the second fall for male 
students and students from rural high schools. Given the number of subgroups examined, however, these impacts 
may not be real because conducting a large number of comparisons increases the likelihood of a result being due to 
chance. See Appendix Exhibit C.9 for more details. 


40 Almost all students (99.6 percent) were sent at least one programmed message. 


41 Advisors excluded some specific messages from being sent to 17 percent of students, based on their 
understanding of these students’ needs. 


42 This particular example explains the dip in the percentage of students who received the fourth message, which 
was a reminder about submitting the FAFSA. One grantee turned off this message for all of its students because it 
knew the students had already submitted the FAFSA. 


43 Students who reported that they planned to delay enrollment until the spring semester were sent a reduced set of 
messages in the fall semester. These messages included reminders about completing the FAFSA for the spring term, 
registering for courses, attending orientation if offered, and paying tuition. During the spring semester, they were 
then sent some of the mindset messages they missed during the fall semester and were also sent the same spring 
messages as students who enrolled in the fall. 


44 The opt-out rate in this study is similar to that in other text messaging studies (Castleman and Meyer 2020; 
Castleman and Page 2016; Page, Castleman, and Meyer 2020). 


45 Lenhart 2015. 


46 By 2016, text messaging was becoming a more common channel to reach college-age students. To determine 
whether students in the study might be receiving similar information via text messaging from their college, the 
study systematically reviewed college websites and called college student support offices. This review documented 
that only three of the 33 colleges enrolling at least 25 study students provided systematic information to students 
through text messaging. 


47 Wrong number responses and opt-outs are not included in the percentage of students replying at least once. 
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48 The exception was Page and Gehlbach (2017), which had a slightly higher percentage of students respond to at 
least one text message (85 percent, compared to 84 percent in this study). Two of the prior studies - by Phillips and 
Reber (2018) and Dobronyi, Oreopoulos, and Petronijevic (2019) - did not report the percentage of students who 
responded to at least one message, though Dobronyi and colleagues note that they did not prompt students to 
respond to the messages and few students did respond. The remaining 11 studies with lower percentages of 
students responding to at least one message than in this study are by Avery et al. (2020); Bird et al. (2019); 
Castleman and Page (2015, 2016, 2017); Castleman and Meyer (2020); Castleman et al. (2017); Mabel, Castleman, 
and Bettinger (2017); Oreopoulos, Patterson, Petronijevic, and Pope (2019); Page, Castleman, and Meyer (2020); 
and Page, Sacerdote, Goldrick-Rab, and Castleman (2019). 


49 On average, students were sent 34 of the 37 programmed text messages. 


50 The word cloud was created using students’ responses to the advisor text message shown in the graphic. A 
standard list of the most common words in the English language (such as a, and, but, I) was consulted to remove 
these words from the responses because their ubiquity in all communication suggests these words have little value 
in understanding students’ interactions with advisor messages. The remaining words that appeared at least 10 times 
across all students’ responses were used to create the word cloud. The size of each word is proportional to the 
number of times it was used in student responses. 


51 In addition to continuing the conversation via text, advisors always had the option to call a student or schedule an 
in-person appointment if they thought that would be best. Therefore, the data on student responses likely 
undercount the level of interaction between advisors and students. 


52 Advisors sent replies to almost all (96 percent) of the subset of students who responded to a message. 
53 Most often, students were sent four non-programmed messages from their advisor. 


54 This information about the regular college transition activities provided to both the messaged and non-messaged 
group was obtained from staff in 11 of the 16 study grantees, at the time transition services were provided. The 
study did not collect systematic information from students about the specific services they received in the year after 
high school. 


55 Bird et al. 2019; Page et al. 2019; Avery et al. 2020. 


56 Other advisors were located at the GEAR UP grantee central offices and local colleges. Information about advisor 
backgrounds came from the study’s “welcome calls.” The welcome calls occurred in the spring before the advising 
began and collected information about each advisor’s location, current position, years of experience in current 
position, advising experience, texting experience, and existing relationship with the students. Calls were not able to 
be conducted with 17 of the 125 advisors. Of the 80 advisors who were located at high schools, 57 had been 


employed at the high school for at least one year prior to the start of the study. 


57 See Appendix Exhibits C.13 through C.16 for more details about how the effects of text-message-based advising 
varied across GEAR UP grantees. 


58 Castleman and Page 2015, 2016, 2017; Oreopoulos et al. 2019; Oreopoulos, Petronijevic, Logel, and Beattie 2020. 
59 Conversation with West Virginia’s GEAR UP program director as part of the study’s advisory panel. 
60 Castleman and Page 2015. 


61 Results shown in Exhibit 8 are for the non-messaged group, as the reference group, but were exactly the same for 
the messaged group. The study’s survey was administered to high school seniors in either spring 2016 (cohort 1) or 
spring 2017 (cohort 2). The survey asked students the following question: “Since the beginning of last school year, 
that is, your junior year, how many times have you discussed the following topics with a college counselor/advisor 
at your high school: (a) College graduation rates, employment rates, and/or other student outcomes at different 
colleges; (b) Admissions requirements (such as SAT/ACT scores, transcripts, and letters of recommendation) for 
different types of colleges; (c) Timelines for applying to college; (d) How to complete the Common Application; (e) 
Your family’s options for paying for college; (f) How much you and your family will have to pay for college if you get 
financial aid; (g) How to complete the FAFSA form; (h) Colleges that would be a good fit for you based on your 
grades, resources, and interests.” Results for all students surveyed, which include students not eligible for the text- 


message-based advising study because they did not plan to enroll in college after high school, are shown in Appendix 
Exhibit E.2. 


62 The national estimates come from questions on the U.S. Department of Education’s High School Longitudinal 
Study Survey that asked students, “Did you meet one-on-one with a high school counselor in the 2012-2013 school 
year [12th grade] about gaining admission to a college or university?” and “Did you meet one-on-one with a high 
school counselor in the 2012-2013 school year [12th grade] about applying for financial aid?” 


63 For more information about the text-based advising offered by the Common Application see 
https://www.commonapp.org/college-advising, and for information about the College Board’s texting program, see 
https://go.collegeboard.org/opportunity-scholarships-student-email-text-message-notifications. 


64 Executive Office of the President, National Science and Technology Council 2015. 


65 Avery et al. 2020; Bird et al. 2019; Castleman and Page 2015, 2016, 2017; Castleman and Meyer 2020; Castleman 
et al. 2017; Dobronyi et al. 2019; Mabel et al. 2017; Oreopoulos et al. 2019; Page et al. 2020; Page and Gehlbach 
2017; Page et al. 2019; Phillips and Reber 2018. 


66 For example, Page, Lee, and Gehlbach (2020) found that text-message-based outreach was most effective for 
discrete, time-sensitive, administrative tasks. Similarly, across many studies over a five-year period, Oreopoulos and 
Petronijevic (2019) found that text-message-based campaigns improved some outcomes, such as students’ study 
time, but did not significantly change any academic outcomes. 


67 Rather than relying on a single study to understand whether text messaging works, an analysis that combines 
evidence from this study with relevant information from prior studies (known in statistics as a Bayesian analysis) 
shows that there is a very high probability that college transition messaging has a small, positive effect on 
immediate college enrollment and a very low probability that college transition messaging increases immediate 
college enrollment by more than 2 or 3 percentage points. This analysis uses results from seven of the 14 prior 
studies shown in Exhibit 9. All studies from this exhibit that examined immediate college enrollment as an outcome 
are included in the Bayesian analysis, discussed in detail in Appendix D. 


68 LaFave, Kelly, and Ford 2018. 
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